15 research outputs found
Deformable Convolutional Networks
Convolutional neural networks (CNNs) are inherently limited to model
geometric transformations due to the fixed geometric structures in its building
modules. In this work, we introduce two new modules to enhance the
transformation modeling capacity of CNNs, namely, deformable convolution and
deformable RoI pooling. Both are based on the idea of augmenting the spatial
sampling locations in the modules with additional offsets and learning the
offsets from target tasks, without additional supervision. The new modules can
readily replace their plain counterparts in existing CNNs and can be easily
trained end-to-end by standard back-propagation, giving rise to deformable
convolutional networks. Extensive experiments validate the effectiveness of our
approach on sophisticated vision tasks of object detection and semantic
segmentation. The code would be released
In-Hand Object Rotation via Rapid Motor Adaptation
Generalized in-hand manipulation has long been an unsolved challenge of
robotics. As a small step towards this grand goal, we demonstrate how to design
and learn a simple adaptive controller to achieve in-hand object rotation using
only fingertips. The controller is trained entirely in simulation on only
cylindrical objects, which then - without any fine-tuning - can be directly
deployed to a real robot hand to rotate dozens of objects with diverse sizes,
shapes, and weights over the z-axis. This is achieved via rapid online
adaptation of the controller to the object properties using only proprioception
history. Furthermore, natural and stable finger gaits automatically emerge from
training the control policy via reinforcement learning. Code and more videos
are available at https://haozhi.io/horaComment: CoRL 2022. Code and Website: https://haozhi.io/hor
Perceiving Extrinsic Contacts from Touch Improves Learning Insertion Policies
Robotic manipulation tasks such as object insertion typically involve
interactions between object and environment, namely extrinsic contacts. Prior
work on Neural Contact Fields (NCF) use intrinsic tactile sensing between
gripper and object to estimate extrinsic contacts in simulation. However, its
effectiveness and utility in real-world tasks remains unknown.
In this work, we improve NCF to enable sim-to-real transfer and use it to
train policies for mug-in-cupholder and bowl-in-dishrack insertion tasks. We
find our model NCF-v2, is capable of estimating extrinsic contacts in the
real-world. Furthermore, our insertion policy with NCF-v2 outperforms policies
without it, achieving 33% higher success and 1.36x faster execution on
mug-in-cupholder, and 13% higher success and 1.27x faster execution on
bowl-in-dishrack.Comment: Under revie
Coupling Vision and Proprioception for Navigation of Legged Robots
We exploit the complementary strengths of vision and proprioception to
develop a point-goal navigation system for legged robots, called VP-Nav. Legged
systems are capable of traversing more complex terrain than wheeled robots, but
to fully utilize this capability, we need a high-level path planner in the
navigation system to be aware of the walking capabilities of the low-level
locomotion policy in varying environments. We achieve this by using
proprioceptive feedback to ensure the safety of the planned path by sensing
unexpected obstacles like glass walls, terrain properties like slipperiness or
softness of the ground and robot properties like extra payload that are likely
missed by vision. The navigation system uses onboard cameras to generate an
occupancy map and a corresponding cost map to reach the goal. A fast marching
planner then generates a target path. A velocity command generator takes this
as input to generate the desired velocity for the walking policy. A safety
advisor module adds sensed unexpected obstacles to the occupancy map and
environment-determined speed limits to the velocity command generator. We show
superior performance compared to wheeled robot baselines, and ablation studies
which have disjoint high-level planning and low-level control. We also show the
real-world deployment of VP-Nav on a quadruped robot with onboard sensors and
computation. Videos at https://navigation-locomotion.github.ioComment: CVPR 2022 final version. Website at
https://navigation-locomotion.github.i